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ABSTRACT 

Presented is a pape r on a strategy for in -house 
evaluations in the context of an educational research and development 
facility. The obstacles in conducting an evaluation ^of colleagues* 
programs are discussed^ and a strategy is proposed which places the 



evaluat'or i 



rcls of a coordinator^ facilitator, and synthesizer 



of several separate research efforts in which colleagues in a variety 
of disciplines combine energies and interests to assess the program, 
being examined. It is noted that recruitment and leadership are 
accomplished by persuasion, and by assuring payoff to the researchers 
involved' in the form of contributions to their own area of research. 
Tactics for organization and manageaent are discussed in some detail. 
Preliminary results from ongoing research projects are seen to 
sug-gest that the evaluation effort was facilitated and i.ts value ^ 
increased by multidisciplinary contributions to the research. 
(Author/SB) 
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^ . . _ Absfrart ^ 

This paper propos€;s a strategy for ui-house evaluations in the con- 
text of an educational research and development facility. The obstacles in 
conducting an evaluation of colleagues' programs are discussed. The pro- 
posed strategy places the evaluator in, the role of a coordinator, facilitator, 
and synthesizer of several separate research efforts ui which colleagues 
in a variety of disciplines combi'ne e^iergies and interests to ass-ess the 
program being examined. ^ecJraitment and leadership were accomplished 
by persuasion and by assuruij^j payoff to the researchers involved in the 
form of contributions to their own area of research. Tactics for organiza- 
tion and management are discussed in some detail. Preliminary results 
suggest that the evaluation effort was facilitated and its value increased by 
multidisciplinary contributions to th^ research. 
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A STRATEGY FOR PROGRAM EVALUAT 



Gaea l^iniiar^ 



Univerftity of Pittsburgh 



In this- paper I w^ll describe a strategy for the in-house evaluation of 
a specific curriculum in the cont<^ of an educational research and develop- 
ment center. The particular ^friculum which is examined is the Individu- 
alized Science (15) curriculum (Champagne Klopfer, 1974), the particular 
research and development center i's the Learning Research and Develop- 
ment-Center (LRDC, Note 1), The need for a new approach to evaluation 
In this context derives from the problems inherent in the role of an evalua- 
tor faced w^^making public judgments of the value of colleagues' work 
^nd from those residing in the need to produce detailed and convincing evi- 
dence lot an evaluation. In order to solve these problems, a ^roce^dure 
^as developed whitfh places the evaluator in the role of coordinator, facili- 
tator, and synthesizer of a cooperative, interdisciplinary r^earch effort 
which brings a variety of research ;*pproaches to bear on a cluster of prob- 
lems. Here I will describe the issues which led to the genesis of this pro- 

Lll t 



cedure and detail the general strategy and tactics which were developed to 



in^rove its (jhan^s of success. Since the effort is on-going, th*re are no 
results in the usual sense. However, th^ undertaking has led to the initia- 
tion of specific research projects whose results will bear on the overall 
evaluation, and these studies aiV^escriioad. • 



Problem ' ♦ 

The problems which confront a researcher engaged in evaluating the 
products of colle^ues in the same organization fall into essentially two 
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categories. First, there is a of problems that relate to conflicts in- 

re nt in jud gin g the wo r k^f_a collea gue without the protection of anonym ity 
ler^r^amzniDnar^utEQ^^ J**" 

posed by the unique characteristic ^ of the program u nder c nn^iderAtion^ 

which can a ppear m ore co mpl ex than the y, re ally are due to the myopia 
induced by such close 'association. The strategy described in this paper 
is designed to turn these apparent disadvantages into advantages in the 
evaluation of educational programs. 

As has been documented by Scriven (1972) and others (Thrasher, 
Frankhn, k Kittredge, Note 2), it is often difficult for researchers in the 
same organization to dejjign and carry out studies which are aimed at 
Evaluating the work of their c^olleagues, especially when the results of 
that research are to be publicly consumable. Regardless of whether such 
research is unbiased and objective, it will inevitably be viewed with suspi- 
cion by outsiders U the results are favorable. If the results are unfavora- 
ble, a storm of protest will be raised about methodology, sensitivity, 
insightfulness, and finally, the value of the evaluation. ^ Surviving such a 
storm is difficult at a distance and debilitating when it takes place in one's 
own back yard. Inevitably, if th*re are posiUv.e results, the "outside" 
world will be suspicious, if there are negative results, the "inside" world 
will be suspicious (see Rossi &f Williams, 1972). 

Possible solutions to this dilemma have been debated in the literature. 
The most popular solution is to remove the evaluative process from the 
scene of developmental effort either by bringing m evaluators who are 
external to the organization or by placing the evaluators in a position of 



See the now classic debate over the evaluation of Head Start between 
Cicirelli and Campbell and Erlebacher in Hellmuth (1970) and a discussion 
of it by Williams and Evans in Rossi and Williams (1972), 

\ 
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higher organizational «.s.*v,ority than the developers (Thrasher et al. , / 
Note 2). Unfortunately, this solution raises more problems than it solves. 
' ^O utsiders ' ' pay for their ability to be critical without consequence by a 
1^53 or knowl«<Jge-and by * limitation on their sources of information 

CR^sTT , 1 9 72)^ Que v ^ a y o ut o f t hrs-e ^und r ^ rr . i t t o ns p the . poteR U&Iior 

increased information, which the in-house researcher cap^^ave, to pro-^ 
vide the most convincing evidence for the value of the prQgram. trade^-^ 
off IS between dubiously credible objectivity and ^ftique accessibility to / 
information. Neutrality. is no guarantee of high qualit^j^^earch; n^ithjer 
does the lack of neutrality insure against it. In fact, effort>-<o prese^e 
so-'called neutraiity can often lead to mediocrity in research b^^6Viding 

unbiased, "clean, " but irrelevant data for evaluation. ^ \ 

\ 

. Evaluation can a<m to be serviceable and waive the nol^al g\^eli*ies 
for research, or it can aim to be reseaiK:h and seek to be serviceable within 
that context. The issue is one of degree. Clearly an^ejeg^t research 
report delivered three years after the appropriate deadline is of no use, 
' On the other hand, a service document filled with theoretical or methodo- 
logical errors does not truly fill the service function; instead, it merely 
/ouds the issues. The advantage of emphasizing the research aspect of 
evaluation is that it permits the researcher to invoke the usual rules of 
evidence of research. The evaluator can then assume a stance of support- 
ing or rejecting the value of a particular product, in the same way a re- 
\searchcr favors a particular solution to a problem and supplies evidence/to 
back up the position. . / 

Emphasizing either the re search "aspects or the service, asp<^ts of 
evaluation leaves untouched additional problems that arise from the fact 
that ^ach new program presents its own set of problems for the researcher. 
For example, the LRDC science curriculum is extremely complex, having 
Ynultiple goals and multiple modes of accomplishing those goals. It is a 
program which not only focuses on increasing cognitive abilities wi^ respect 
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to science, but whrch also attempts to teach affectiv/; attitudinal com- 
petencies. Further, the eociccy which examines, purchases, and uses 
the program does not have clear-cut goals for elementary science educa- 
'Hon: IFfhe goals are complex or if the consumers have not statJtd with 

gli^ity or^on-sensu^s whar tKey want "fji^..a,a. e]Vm.patax y..sr lear^ pxog ra m , - ^ 

it is extremely dUficult to provide, e.vidence .that d(^cuments the value of 
the program. " ~ ^ /L 

\unfortunately, mWh of the wprk that has been done in the evaluation ' 
of eduAtional program/s by external evaluators has ignored the issue of 
providing useful inf or^atloa and has instead focused on a set of side issues, 
such as experimental versus quasi- or non- experimental designs in educa- ^ 
tion., Unless treatments arc randomly assigned to the unit of analysis, 
truly comparable "control" groups are almost nev«r available in educa- . 
tional settings. Therefore, the ensuing debate over the results of so- 
called experimental studies tends to shift from an analysis of the progrj 
and its effects to an argument over 'whether Group A was the s^me 
- .equivalent to, or different from Group B {Hellmuth, 19*70). ,'n></observa- 
tion IS not meant to imply a rejecticfn of experimental desj^for evaluation, 
U IS rather an attempt to keep the real problem in foci^nd to suggest that 
valuable information is still obtainable through the *^r^>U^a t. o n o f d ifferent ^ 
techniques. ' * ^ 

In summary^, several problems beset any effort to conduct evaluative 
re/earch in education. The role of the, evaluator can be a source of tension 
'for the evaluator and the organization. . The specif ic/^rog ram under investi- 
;jation always contributors its own- set of difficulties. Finally, problems of 
acceptable and realistic methodologies abound. In the remainde^r of this 
paper I will examin*- one approach to^olving some of Ihes^ problems, 

/ 

/ 
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Resources 

Before detailing the strategy, it is useful t^jj^eirtfribe briefly the 
#emftg^^nd th^ av^Uable refeources in wh^p/ftbp. .strategy wa.s dp.velnpp.dl — 

The Learning Research and Development Center is>one of 14 feder- 

ally funded educational laboratories andjcenters in the United States. LRDC 
is devoted to the impfovefnei« Df gducittoijjiniagh liidiyidOT — - 

adapting the content, and pace of instruction to the individual needs of the 
student (Glaser, 1973). It consists of 23 research projects, the majority 
of which report directly to one of three Center director's. Each project ^ 
has between one and three Ph. D level people and a support staffs The 
Center*lias attracted a number of 'prof essional educators, psychologists, 
and sociologists, all of whom work on diverse topics which tend to con- 
verge in their purpose if not in their approach. For the evaluation task, 
a subs.et of that community has been drawn together. The subset consists 
of curncuium developers, evaluators, and some individuals Jess directly 
and less obviously related to evaluation who are engaged in basic research 
in psychology, anthropology, and educational research. 

The other resources available for the evaluation'effort consist of 
three local schools which are using the science curriculum. Each school 
has at least one science classroom in which the. Individiialized Science pro- 
gram IS being used. Each science room services between five and eleven 
classes of science. The three schools range in the degree to which the 
program is implemented and the degree to which the surrounding environ- 
ment supports individualized education. Two of the settings are develop- 
mental public schools^ w^th several individualized prog/ams in operation, 
and the third is a parochial school which has no other formally individual- 
ized curricula. Although there is no immediate access to "control" Schools, 



2 * - 
Developipental public schools are schopls with which LRDC has a con- 
tractual relationship permitting curriculum development work to be conducted. 




our experience has been that limited amounts of research carf be conducted 
at other schools in the Pittsburgh area. Limited research can be carried 

out when a Urg < ?. s c a l e o if^t in v oW irt g t ea ti n g and o ther disru p tiv e .ulOx^- 

yvention would not be acceptable 'but ^maller^ -^hor -t- te^rm-^on^ymon th )-inYe&— ' 
Hgatibns would be. ' ' 



/ 



A Strategy 

. ' ^ . 

In order to effectively use the , resources available in the Center^ to 

provide evidence for .the Value of the science p^rogram and to ^ivoj^d as many 
of the drawbacks as pods^l^le, the schema represented in Figure 1 (pa'ge 11) 
was developed. The figure diagrammatically illustrates how specific re- 
sources were brought to bear in solving the problem of establishing the 
value of the rndividualizeJ Stie^nce cXirriculum. ^ The basic strategy was 
to bring together the diverse abilities and interests extant in the Center to 
investigate various aspects of the science curriculum wfjile simultaneously 
offering re'searchers an opportunity to contribute'in any way they wished to 
their own problem areas. Initially* the schema was worked out with only 
vague indications of peripheral specializations. It was not expected that all 
areas of the science program woul^d.be studied, nor was it expected that 
each area would be studied in equal depth. What was exp^ected was tha\ the 
collection of studies from differing perspectives would contribute greatly 
to the available knowledge about the wajrXn^which this specif ic 'prjog ram 
works. There is, of course, j\o guairailtee that the knowledge thus generated 
will be totally adequate to meet the needs of educational consumers, but ti-a- 
ditional approaches to evaluation cannot guarantee that either. ^ 



Obviously, this entire ^ffort depfends upon both the cooperation and 
enthusiasm of the entire Scien,ce .Project. Without their generous support, 
none of the studies could h^ve b'een conducted. « 1 



ERIC /■ . 9 



Tactics 

To dperationahze thi? strategy, tactics had to be Geveloped to attract 
rp«^<>^rch<>ys i^ ^om different ^> roups, to provide circu mstances in whic h they 



could continue to work on their c^wrx areas while contributing to the science 
revaluation, and iinally, ta^^ievelop-zTsettlng Lir-wrtmrir niea^-^it:!^ w<^r^ot - - 
fully formed could be aired and improved^pon. Tlie tactrcs used to i mple- 
ment this strategy fall into roughly two categories: those behaviors asso- 
ciated with initial selection and recruitment of researcher s, , and those 
^behavior^ associated with development and maintenance of. a task perform- 
mg group. 

In theory, an optimal selection of disciplines to be repr esente^Ti^^ 
the peripheral specialties) c(^ld be determine<i a priori and jieople assigned 
to do specific projects so that urti<;^ue iniormation would be maximized while 
oVerlap would be minimized. In reality, however, while disciplines and 
research areas could be selected in advance, people could not* be</^u'se 
LRDC IS a semi-hxerarchlcal organization rather than a hierarchical one. 
Leadership is by persuasion rather tkan by cc^r?Tmand. ThereWre, research- 
ers who were both available and in a relevant area had to be Attracted to the 
task and then worke'd with -to develop studios which maxirpizf^d the informa- 
tion eagh could contribute. ' - 4 

Selection and recruitment . Initially, a list of individuals who might 
be (interested ui working on the science evaluation effort was drawn upv- , 
Some individuals were included because they were connected with the 
Science ^or Evaluation Projects. Others were added because their work 
or work.interests indicated thkt their inclusion w^ould be mutually benefi- 
cial. Still others were recommended or volunteered o^t of personal inter- 
est as the initial meetings took place. Several graduate students v|ere. 
. included on the list because it was assurped that the </pportunity b6th to 
obtain thesis material and to develop pilot studies it^^a focused problem 
setting would be attractive^ to^them. / / 




iU 



* After this list was constructed, Individualis were contacted one at a 
time and the basic problems were discussed. On the one hand, the science 
pro gram and the problem of ev a^atlng it had to b£Jnaiie_,iati:ig-Uing^.,j;Lfii&^__ 
vant, and seductive. On the other hand, payoff to the individuals involved 
in terms of th^ir own basic research interest had to be relatively assured. 
In other w&r^, jjt-was Important for tire prod\^c ts ^3^*^f*«earch/^contrribTite 
to both the evaluation problem and the s*pecific discipli^^©.s-,^-*^^epresented 
by the double-headed arrows m Figure 1 (page 11). ^It was felt that given 
access to "natu^l" settings, many researcjiers would welcome the chance 
to put some oT their work to applied use^. The r^jpnaini^g task was to pro- 

ide a foryrn in which proposals for^research and initial results could be 
dl8&u8»^d in a supportive wayj. / 

• / ' V 

Maintenance . Small group meetings, which included t^iOse /esearch- 
^^rs ^ho Ayere already "on board, " were held once 6very two weeks,^ while 
the individual meetings to recruit additional researchers were being con- 
ducted. Several decisions were made to irigrease the probability of the 
meetings', success/ They were to bp -open to interested researchers, how- 
ever, an effort was ma^de to keep the proportion of active working partici- 
pants to observers high. Individuals were encouraged to attend nrv^feetings 
or send representeitives to see what they were about, but they were dis- 
couraged from continuing to attend if they did not intend to do any research 
in the area. Ground rules for discussidn'of research proposals were laid 
down which encouraged directed, constructive criticism and discouraged 
criticism for its own sake. 



The purpose of the meetings was to discuss bile by pne each of the / 

proposed or in-process studies. The discussions focused on the relevance, 

feasibility, and significfinfce of each study. In some cases the discussion 

pointed out so marty prqblerhs with a particular proposal that the study was 

J . ^ ^ ' _ , 

abandoned. In other casesr discussions wande^red away from the initial task 

o? examining* pr<3po»als t^ excursions into what evaluati(j)n should or could be. 



EMC 



1.1 



/ 



•/ 



f 



These wanderings were riot '^^Wether purposeless for they helped to clarUy 
what the type and e^r^t of the^esul^knt wiformation would be. Along with 



the ^ejafeC-alr^^<<r»ioni_ fPA^^I^i^ yethods for implementing each study- in. 
field -sett^gs welre sketched out! 

TK^TT^re drawback^ associated withlhis process. ±5y naving 
-rc5tr«rcherTf rom so many areas aU^tryg^lt^-^vorK^o^^^^^^ 



arose concerning their individual roles'. For example, some researchers 
attc^pte^ to become evaluators and entirely abandoned their own disci- 
plinary perspectives rather than merging the two. Further, because the • 
majority of the researchers were neither evaluators jior science educators, 
problems arose over the n<ied to familiarize everyo/ie with science consent, 
placing a burden on the Science Project Staff. F^lfially, the most serious 
risk lay in the potent^ial of producing disparate fragmented studies Which ' 
would have no common theme or insight. However, the majority of these 
. difficulties were overcome by continuous discussion in the^glroup meetings 
during the first two or three months c^f initial work. 

' During the first sev'en-mbntK period, 13 meetings were held. The 
first meetings were largely organizational and introductory. In these meetv 
ing.s the evaluators functioned as facilitators; they did not attempt to direct 
the kind of research to b«, conducted* As the meetings progres'sed, ^ for- ' 
.mat developed of pre^/^f^TTnTone or bwo proposals for research or reports ^ 
of work in progress per meeting. Each proposal was PWiewed and dis- 
cussed N^ith regard t^both its substance and its feasibility.' It would be a 
legitimate criticism of this approach that not enough initial guidance was 
given, so that discouraging false starts'were sorKetimes made. ^H^wever, 
It was f^t that the error. should be in the direction of too little^rather than, 
too raUch directiven/ss, as this' was cohsi«tent with the; styK,o£_the or^i- 
zayi^n. / ... , , ' , ^ 

' ' Five monthXjfter the meeUngs began, an interesting interruption' 
/occurred. Research^ for Better fechools (RBS) Requested a joint meeting ^ 
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to exchange ideas on the evaluation o£ the science curriculum since they 
were engaged m evaluating the IS field test units as these became availa- 
ble. Their evaluation effort was 6i the more traditional outcome assess- 
ment type. The importance of the meeting for us was that it forced our 
group to summarize exactly what we were doing anH why. This, in turn, 
led to a substantial increase \n group solidarity. 

In summary, recruitme/.. v. as by invitation and not coercion, 
tasks or roles were assigned, they were selected. The maintenance of 
interest in ihe task was accomplished through highly task-oriented arnd 
reasonably supportive group meetings. 

Results to Date 

Figure 1 shows the ten basic studies which currently maKe up the 
group oi studies to be done m science. The specific titl« of e<2Cn study 
appears m italic type on one sid^^i the double-arrowed line'; .the primary 
discipline from which the individual comes is in boldface type on the other 
side. The area other than science evaluation to which the research con- 
tributes appears around the outside periphery of the figure. Trie ro}^ of 
the evaluators is to help facilitate the- research and. after all the studies 
are complete, to coordinate results mto a series of docun^nts that can 
serve evaluative functions. 

Turning to the details of Figure 1, two of the evaluators are int>er- 
<:9ted m validating the mocfel that had recently been proposed for investi- 
gating classroom processes fCooley i Lohnes,* m press). One aspect of 
that model conc<: rns ^he^us e of time and its-effect on student achievement. 
They investigated the relationship between the number of minutes spent in 
science. iV^.ath. and reading, and student achievement m one of the schools 
in which thfe science program existed {Leinhardt ir Cooley, Note 3). -Pre- 
hmmary results indicate that the amount of time spent in science, indepen- 
dent of entering abilities, is positively related not only to science achieve- 
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ment. but also to math and reading achievement. This was supp»Drted by a 
similar clinical finding previously reported by teachers ai)d administrators 
using the science curriculum. This finding has important implications for 
the value of elementary science education to improve not only the standard 
of scientific literacy, but also the overall level of knowledge of elementary 
school children. A replication of this study is currently being conducted. 

O^.e of the social psychologists is interested in studying teacher and 
student perceptions in the classroom. The planning for this research was 
completed before the rr.eetings were started, hbwever, the meetings were of 
some help in implementing the pjans. This researcher is planning a study 
in th« science rooms to investigate student perceptions of learning, school, 
and themselves (Gre^nberg, Note 4). He is examining the students' atti- 
tudes and beliefs about knowledge acquisition and the roles of teachers and 
peers in different environments. This study is being conducted in ap mner- 
city school and will provide information on how children view different learn- 
ing settings and will shed light on how those views might affect learning 
Itself. 

Another social psychologist is interested in the problems of locus of 
control and of changes in perceptions which occur in different educational 
environments. He is conducting a study of locus of control withm the con- 
texts of the science and mathematics classrooms at two grade levels m a 
parochial school. One section of mathemaUcs is taught in a traditional and 
noniridivadualized manner, while science is taught m a more open and indi- 
vidualized way. He is working toward the description and measurement of 
locus of control which is situation specific rather than using theytraditional 
view of It as an underlying trait or characteristic. 

The science educators, who have long been concerned with the prob- 
lems of formative evaluation and its docvmientation, are working with 
sociologists who are interested m documienting variations m the frames of 
refe rence of actors engaged in formative evaluation. The sociologists are 
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ronHur^ipa a r;^se «tudy of the behavicrs cf the developers during a stage 
of curricalurr. development. The product of this joint effort will be a forma 
tive evaluation of a specific unit of the science curriculum and a detaUed 
description of how design decisions are made and by whom. Further infor- 
mation about how the different groups involved with dissemination (publish- 
ers and school personnel) impact curriculum development will also be docu- 
mented. 

Two researchers are interested in developing methodologies for ana- 
lyzing curricula. One ot the two is interested in developing models of 
hierarchy analysis. She is detaUing the implied and overt hierarchies of 
both the affective /attitudinal and the scientific literacy goals of the cyrricu- 
lum. The results of this will be a map of the science curriculum's objec- 
tives and the specific activities designed to meet these objectives. The 
second team of researchers iff interested in developing measures of how 
closely curricula use known behavioral principles. These mV^sures have 
been applied to several diverse curricula (HoUand, 1975) and a\etailed 
analysis of one of the Colter's other curricula has already been c^ducted 
(Holland Solomon, 1975). They are doing a similar analysis of th\ sci- 
ence curriculum. * \ 

One researcher is doing a small study of the science goal of student 
self,-evaluatron. She is interviewing students who have worked through a 
^science unit before they take their posttest for the unit. In the interview 
the students are asked to estimate how much they know abi^f^i given sub- 
area. Their estimation is then compared to their posttest performance. 
In general, students tend to slightly underestimate their knowledge, with 
better students being closer in their estimates than poorer students. The 
importance of this study is that it will help to document the science activi- 
ties speciirically designed to contribute to students' abilities to evaluate 
themselves, and it will also indicate how other curricula not so designed 
impede students' ability to evaluate their own knowledge states. 
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Two other sttidtV^ have recently emerged in the science meetings. 
One IS an ethnographic d^cription of student free-range behavicxrs in the 
science room- The other i^he development of new and more extensive 
, science competency measur/=s hvf elementary and secondary schools 
(labeled "Measurement'; in FigurXj^ Currently, most Qf the standard- 
ized tests do not distinguish between^^^nce and social studies until the 
' fifth grade. Further, the majority of exiHmg tests are so imprecise and 
inaccurate in tHcir language as to place a kY^owledgeable student at a dis< ^ 
tinct disadvantage. 

In summary, these studies contribute to the UQ^erstanding"of IS and 
hence help to show the value of it alofig three damensio>iS. First, reasona- 
bly complete documentation of^the content, pedagogical sound^^ss, jvnd 

^* ^^^^^^ ^ ' 

design decisions of the program will be made available. Second, ^^e > 

insight about specific classroom processes and their impact, both psy'clPib^^ 

logical and social, will be ge^^^rated. Third, the impact of science educa-vf^ 

Uon on both science and o*her basic skill areas will be better understood. '\ 

* ' "> ^ 

' Conclusions ' ' ^ • 

An additional result of this exercise has been the validation of the 
. \ strategy presented^n Figure 1. Figure 2 represents a restatement of that 
^ s:-ateg^y independent of the specific Studies being conducted. At the center 
of Figure 2 is the object of the evaluative inquiry. 'While the object of 
inquiry is singular, the inquiries are considered plural. That is, there is - 
't^ a set of problems or topics for investigation. Around the periphery of 
' • Figure 2 are several problem areas'of research, the focus of w^ich is not 
' necessarily evaluative research, l^ese peripheral foci are areas of 

r'escarch which grow out of specific discipline - related inquiries. The figure 
indicates that the problem areas and related disciplines can be, but are not ^ 
necessarily, mdependcnt^of one another. The strate^is to incorporate the 
work of individuals from diverse backgrounds who are working on problems 
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Figure 2 A stcategy ^or program evaluation. 
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in their own discipline areas into an overall program of evaluative research. 
These individuals represent a powerful and generally untapped resource for 
solving many evaluative research problems. 

The outcomes of this on-gomg effort are encouraging. If- onevwiehes, 
one can exploit the available resources of a resear^t cente r t'^ facilitate 
in-house evaluation in a creative fashion resulting in unique aKd us^ bte 
information. The role oi the evaluator is to orchestrate the undertaking ^ 
and. when the studies are compfieted, to incorporate them tnto an evalua- 
tive document. The product of such an effort can be an in-depth view of 
the program under investigation, one which is richer in expertise and infor- 
mation than any single effort could hope to be. 

This strategy was designed to solve several problems: the conflict 
inherent in the role of in-house evaluator while providing convincing evi- 
dence for the' value of a program and for the soundness of the evaluation, 
and the provision of m<jre chan superficial mformati-on about a highly com- 
plex program. The conflict of the role of the evaluator was solved in two 
ways, first, by sharing the burden of evaluation among several researchers, 
and second, by including the developers in the group. This assured the 
developers that misinformation and misinterpretation of information about 
their progratn could be minimized. The problem of bias is not totally 
solved, however, it ^6 reduced by using a group of people \^rhose primary 
commitment is to good science in their respective disciplines. The problem 
of providing in-depth information ab6ut the program was solved by having 
multiple, and in some cases, overlapping studies conducted ort various facets 
of the program and by making use of already existing resources, many of 
which were previously untapped. 

An additional significance of this strategy for evaluation should not be 
overlooked. One of the major complaints ^of social scientists (such as Rossi) 
i^ that the evaluation field has been unable to attract highly quediiied indi- 
viduals and is the refuge of dropouts from other fields' (Rossi & Williams, 
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1972). The strategy described here provides quality researchers from 
other fields with an.incentive to undej^^^tl^e work that serves evaluative 
functions. Of course, evalu^'r'^^ust do research themselves, but they 
also must continuously encourage their colleagues who are not evaluators 
to engage in research that can also serve the function of evaluation. Not 
only di>«s this provide, one with a sense of parsimony, it also educates and 
^educates evaluators by exposing them to the differing perspectives and 
methodologies of their colleagues, A remaining question is whether or not 
''there are other research organizations that have sufficiently diverse person- 
nel to utilize this approach. I think there are. For example, not only are 
there the university- related research centers, there also are governmental 
institutes and pi^i^ate industry laboratories, all of which have sufficiently 
diverse professional staffs to^permit utlization of this approach. 

The strategy is not offered as a panacea for all the problems that 
face evaluation. It is offered as an alternative to the traditional in-house, 
one- shot studies conducted by a sidgle team of evaluators. There is no 
iron-clad assurance that Answers to a collection of questions generated by 
another group of researchers would provide adequate information about the 
value of a program. However, the strategy provides^mple opportunity for 
the evaluator and other researchers to discuss this point and try to con- 
struct a series of studies that will provide usable ^information for consumers. 
The emphasis in this strategy is for researchers to satisfy themselves with 
regard to the scientific excellence of their work and to seek answers to 
questions which they generate rather than attempting to second-guess the 
consumers' questions. 
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